Dynamic Test Generation with Static Fields and Initializers

Abstract. Static state is common in object-oriented programs. How-ever, automatic test case generators do not take into account the po-tential interference of static state with a unit under test and may, thus, miss subtle errors. In particular, existing test case generators do not treat static fields as input to the unit under test and do not control the execu-tion of static initializers. We address these issues by presenting a novel technique in automatic test case generation based on static analysis and dynamic symbolic execution. We have applied this technique on a suite of open-source applications and found errors that go undetected by ex-isting test case generators. Our experiments show that this problem is relevant in real code, indicate which kinds of errors existing techniques miss, and demonstrate the effectiveness of our technique.

Collaborative Verification and Testing with Explicit Assumptions

Many mainstream static code checkers make a number of compromises to improve automation, performance, and accuracy. These compromises include not checking certain program properties as well as making implicit, unsound assumptions. Consequently, the results of such static checkers do not provide definite guarantees about program correctness, which makes it unclear which properties remain to be tested. We propose a technique for collaborative verification and testing that makes compromises of static checkers explicit such that they can be compensated for by complementary checkers or testing. Our experiments suggest that our technique finds more errors and proves more properties than static checking alone, testing alone, and combinations that do not explicitly document the compromises made by static checkers. Our technique is also useful to obtain small test suites for partially-verified programs

JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction

Popular software testing tools, such as JUnit, allow frequent retesting of modified code; yet the manually created test scripts are often seriously incomplete. A unit-testing tool called JWalk has therefore been developed to address the need for systematic unit testing within the context of agile methods. The tool operates directly on the compiled code for Java classes and uses a new lazy method for inducing the changing design of a class on the fly. This is achieved partly through introspection, using Java’s reflection capability, and partly through interaction with the user, constructing and saving test oracles on the fly. Predictive rules reduce the number of oracle values that must be confirmed by the tester. Without human intervention, JWalk performs bounded exhaustive exploration of the class’s method protocols and may be directed to explore the space of algebraic constructions, or the intended design state-space of the tested class. With some human interaction, JWalk performs up to the equivalent of fully automated state-based testing, from a specification that was acquired incrementally

Inferring Loop Invariants using Postconditions

One of the obstacles in automatic program proving is to obtain suitable loop invariants. The invariant of a loop is a weakened form of its postcondition (the loop's goal, also known as its contract); the present work takes advantage of this observation by using the postcondition as the basis for invariant inference, using various heuristics such as "uncoupling" which prove useful in many important algorithms. Thanks to these heuristics, the technique is able to infer invariants for a large variety of loop examples. We present the theory behind the technique, its implementation (freely available for download and currently relying on Microsoft Research's Boogie tool), and the results obtained.Comment: Slightly revised versio

Residual investigation: Predictive and precise Bug detection

We introduce the concept of residual investigation for program analysis. A residual investigation is a dynamic check installed as a result of running a static analysis that reports a possible program error. The purpose is to observe conditions that indicate whether the statically predicted program fault is likely to be realizable and relevant. The key feature of a residual investigation is that it has to be much more precise (i.e., with fewer false warnings) than the static analysis alone, yet significantly more general (i.e., reporting more errors) than the dynamic tests in the program's test suite that are pertinent to the statically reported error. That is, good residual investigations encode dynamic conditions that, when considered in conjunction with the static error report, increase confidence in the existence or severity of an error without needing to directly observe a fault resulting from the error. We enhance the static analyzer FindBugs with several residual investigations appropriately tuned to the static error patterns in FindBugs, and apply it to nine large open-source systems and their native test suites. The result is an analysis with a low occurrence of false warnings (false positives) while reporting several actual errors that would not have been detected by mere execution of a program's test suite. © 2014 ACM

Optimized Execution of Deterministic Blocks in Java PathFinder

Java PathFinder (JPF) is an explicit-state model checker for Java programs. It explores all executions that a given program can have due to different thread interleavings and nondeterministic choices. JPF implements a backtracking Java Virtual Machine (JVM) that executes bytecodes using a special representation of JVM states. This special representation enables JPF to quickly store, restore, and compare states; it is crucial for making the overall state exploration efficient. However, this special representation creates overhead for each execution, even execution of deterministic blocks that have no thread interleavings or nondeterministic choices. We propose mixed execution, a technique that improves execution time of deterministic blocks in JPF. Our technique leverages the fact that JPF is written in Java: JPF is a special JVM that runs on top of a regular, host JVM. Mixed execution works by translating the state between the special JPF representation and the host JVM representation. We also present lazy translation, an optimization that speeds up mixed execution by translating only the parts of the state that an execution dynamically depends on. We evaluate mixed execution on six subject programs that use JPF for generating tests for data structures and on one case study for verifying a network protocol. The experimental results show that mixed execution can improve the overall state exploration time up to 36.98%, while improving the execution time of deterministic blocks up to 69.15%

H-Fuzzing: A New Heuristic Method for Fuzzing Data Generation

Part 1: Session 1: Filesystems and DataInternational audienceHow to efficiently reduce the fuzzing data scale while assuring high fuzzing veracity and vulnerability coverage is a pivotal issue in program fuzz test. This paper proposes a new heuristic method for fuzzing data generation named with H-Fuzzing. H-Fuzzing achieves a high program execution path coverage by retrieving the static information and dynamic property from the program. Our experiments evaluate H-Fuzzing, Java Path Finder (JPF) and random fuzzing method. The evaluation results demonstrate that H-Fuzzing can use fewer iterations and testing time to reach more test path coverage compared with the other two methods

Symstra: A framework for generating object-oriented unit tests using symbolic execution

Abstract. Object-oriented unit tests consist of sequences of method invocations. Behavior of an invocation depends on the method’s arguments and the state of the receiver at the beginning of the invocation. Correspondingly, generating unit tests involves two tasks: generating method sequences that build relevant receiverobject states and generating relevant method arguments. This paper proposes Symstra, a framework that achieves both test generation tasks using symbolic execution of method sequences with symbolic arguments. The paper defines symbolic states of object-oriented programs and novel comparisons of states. Given a set of methods from the class under test and a bound on the length of sequences, Symstra systematically explores the object-state space of the class and prunes this exploration based on the state comparisons. Experimental results show that Symstra generates unit tests that achieve higher branch coverage faster than the existing test-generation techniques based on concrete method arguments.